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Ultimate "SIR" in Autonomous Linear 
Networks with Symmetric Weight Matrices, 
and Its Use to Stabilize the Network - A 
Hopfield-like network 

Zekeriya Uykan 



Abstract 

In this paper, we present and analyse two Hopfield-like nonlinear networks, in continuous-time 
and discrete-time respectively. The proposed network is based on an autonomous linear system with a 
symmetric weight matrix, which is designed to be unstable, and a nonlinear function stabilizing the whole 
network thanks to a manipulated state variable called"ultimate SIR". This variable is observed to be 
equal to the traditional Signal-to-Interference Ratio (SIR) definition in telecommunications engineering. 

The underlying linear system of the proposed continuous-time network is x = Bx where B is a 
real symmetric matrix whose diagonal elements are fixed to a constant. The nonlinear function, on the 
other hand, is based on the defined system variables called "SIR"s. We also show that the "SIR"s of 
all the states converge to a constant value, called "system-specific Ultimate SIR"; which is equal to 
j-^ — where r is the diagonal element of matrix B and Xmax is the maximum (positive) eigenvalue of 
diagonally-zero matrix (B — rl), where I denotes the identity matrix. The same result is obtained in 
its discrete-time version as well. 

Computer simulations for binary associative memory design problem show the effectiveness of the 
proposed network as compared to the traditional Hopfield Networks. 
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Index Terms 

Autonomous Continuos/Discrete-Time Linear Systems, Hopfield Networks, Signal to Interference 
Ratio (SIR). 



I. Introduction 

Hopfield Neural Networks [1] has been an important focus of research area in not only 
neural networks field but also circit and system theory since early 1980s whose applications 
vary from combinatorial optimization (e.g. [2], [3] among many others) including travelling 
salesman problem (e.g. [4], [5] among others) to image restoration (e.g. [6]), from various control 
engineering optimization problems including robotics (e.g. [8] among others) to associative 
memory systems (e.g. [7], [11] among others), etc. For a tutorial and further references about 
Hopfield NN, see e.g. [10] and [9]. 

In this paper, we present and analyse two Hopfield-like nonlinear networks, called SALU-SIR 
and DSALU-SIR in continuous-time and discrete-time respectively. 

The proposed networks consist of linear and nonlinear parts: An autonomous linear system 
with a symmetric weight matrix, which is designed to be unstable, and a nonlinear function 
stabilizing the whole network. The underlying linear system of the proposed continuous time- 
network is 



X = Bx (1) 

where x shows the derivative of x with respect to time, i.e., x = ^, and B is called system 

matrix or weight matrix. 

We define the following system variables denoted as 6i, 

m-j-Jp^---,, i-i,...,N (2) 

where Xi{t) represents the i'th neuron and bij is the ij'th element of matrix B. In this paper, 
we assume di — for the sake of brevity. 

We observe that the manipulated state variable 9i{t) of eq.(2) resembles the well-known Signal- 
to-Interference-Ratio (SIR) definition in telecommunicaions engineering, which can be found in 
any related texbooks: Signal-to-Interference Ratio (SIR) is an important entity in commucations 
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engineering which indicates the quality of a link between a transmitter and a receiver in a multi 
transmitter-receiver environment (see e.g. [13], [18]). For example, let represent the number 
of transmitters and receivers using the same channel. Then the received SIR is given by (e.g. 
[13]) 

7^{t) = ^Jf^ 7-y ^ = l,...,N (3) 

where Pi is the transmission power of transmitter i, g^j is the link gain from transmitter j 
to receiver i (e.g. in case of wireless communications, gij involves path loss, shadowing, etc) 
and Vi is the receiver noise at receiver i. Typically in communication systems like cellular radio 
systems, every transmitter tries to optimize its power pi such that the received SIR (i.e., 7^) in 
eq.(3) is kept at a target SIR value, 7*^*. 

It is well-known that the eigenvalues of the system matrix solely determine the stability of 
a linear dynamic networks. If any of the eigenvalues is positive, then the system is unstable. 
Designing the linear part of the network to have positive eigenvalue(s), we show, in this paper, 
that the defined "SIR" in eq.(2) for any state appraches asymptotically to a constant, called 
"ultimate SIR", which is a function of the diagonal entry of the system matrix and its maximum 
eigenvalue. The nonlinear part of the network uses this result to stabilize the system. Finally, 
the proposed network is shown to exhibit features which are generally attributed to Hopfield 
Networks. Taking the sign of the converged states, the proposed network is applied to binary 
associative memory systems design. 

The paper is organized as follows: The "ultimate SIR" is analysed for the underlying linear 
dynamic part of the network with a symmetric weight matrix in section 11. Section HI presents 
the stabilized network by the Ultimate "SIR" to be used as a binary associative memory system. 
Simulation results are presented in section IV followed by the conclusions in Section V. 

II. Ultimate "SIR" Analysis 

In this section, we analyse the underlying linear system of the proposed network in both 
continuous-time and discrete-time respectively. 
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A. Continuous-time Analysis 

In this paper, we examine the case where weight matrix B in eq.(l) is a real symmetric matrix 
whose diagonal elements are equal to a constant. So, the linear system in (1) can be written as 
follows 

x=(-rl + w)x (4) 

where 



rl 



W 



r 
r 





W21 






r 



and 



Wl2 





Win 

W2N 



(5) 



wm wn2 ■■■ U 
where Wij = Wji, i,j — 1,2, . . . ,N . 

In eq.(5), the design parameter r, which corresponds to ha in the "SIR" (6'j) definition in 
eq.(2), is positive, r > 0. For the sake of brevity, we assume that the 6j = 0, i = 1,2, . . . ,N 
in the analysis. 

It's well known that desigining the weight matrix W as a symmetric one yields that all 
eigenvalues are real (see e.g. [16]), which we assume throughout the paper for the sake of 
brevity of its analysis. 

Proposition 1: 

Let's assume that the linear dynamic network of eq.(4) with a real symmetric W in (5) has 
positive eigenvalue(s). If r > is chosen such that it's smaller than the maximum (positive) 
eigenvalue of W, then 

1) the defined "SIR" {9i) in eq.(2) for any state i asymptotically converges to the following 
constant as time evolves for any initial vector x(0) which is not completely perpendicular 
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to the eigenvector corresponding to the largest eigenvalue of W. ^ 

QUlt ^ QUlt ^ i = l,2,...,A^ (6) 

where \max is the maximum (positive) eigenvalue of the weight matrix W. 
2) there exists a finite time constant for a given small positive number ec > such that 

> M -^""''"ll < ec (7) 
where O^^^t) = [^i(t) . . . ^jv(t)]^ and ^"^^'^^'^ = . . . 1]^ and 1 1 • 1 1 shows a vector norm. 
Proof: 

Defining the following matrix series equation 

fe=0 ^' 

{-rI + W)H^ (-rl + W)H^ 
^ 1! ^ 2! ^ 

— r^+--- 

where (!) shows the factorial, and I represents the identity matrix of right dimension, it's well 
known that the solution of the autonomous dynamic linear system in eq.(4) is (see e.g. [17]): 

x(t) = e(-'''+^)*x(0) (10) 

where x(0) shows the initial state vector at time zero. So, the state transition matrix of the 
linear system of eq.(4) is e(~''^+^)* in eq. (8) (see e.g. [17]). 

Let us first examine the powers of the matrix (— rl + W) in (8) in terms of matrix rl and 
the eigenvectors of matrix W. First let's remind some spectral features of the symmetric real 
square matrices that we use in the proof later on: 

It's well known that any symmetric real square matrix can be decomposed into 

' It's easy to check in advance if the initial vector x(0) is completely perpendicular to the eigenvector of the maximum 
(positive) eigenvalue of W or not. If this is the case, then this can easily be overcome by introducing a small random number 
to x(0) so that it's not completely perpendicular to the mentioned eigenvector. 
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N 

W = ^A,v,vf (11) 

i=l 

where {Xi}f=i and {vi}^^^ show the (real) eigenvalues and the corresponding orthonormal 
eigenvectors (see e.g. [16]), i.e.. 



vfv, = <( •' " ^ ' (12) 



1 if and only if i = j 
otherwise, 

where i,j — 1,2,. . .,N. Defining the outer-product matrices of the eigenvectors {Aj}^i 
as Vj = Vj vf in eq. 1 1 gives 

N 

W = J2>^i^i (13) 

i=l 

The matrix (— rl + W) can be written using eq.(ll) as follows 

TV 

-rI + W = -rI + ^A(l)Vi (14) 

i=l 

where r is the diagonal element of matrix rl, and where Pi{l) is equal to 

A(l) = A, (15) 
The matrix (— rl + W)^ can be written using eq.(ll)-(13) as 

N 

(-rl + Wf = r^I + A(2) V, (16) 



i=l 



where Pi{2) is equal to 



A(2) = -rA, + (-r + A0/3(l) (17) 
Similarly, the matrix (— rl + W)^ is obtained as 



N 

+ w)=^ = -rn + A(3) V, (18) 

i=l 



where /9i(3) is 
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For A; = 4, 



Pi{3)^r^Xi+{-r + Xi)P{2) (19) 



AT 

-I + W)^ = /I + ^A(4)V, (20) 
1=1 



where A (4) is 



A(4) = -r3A, + (-r + A,)/3(3) (21) 
So, when we continue, we observe that the k'th power of the matrix (— rl + W) is obtained 



as 



N 

(-rI + W)^ = (-r)'=I + X;A(A;)V, (22) 

i=l 

where for /c > 2 is equal to 

Pi{k) = {-r)''-'Xi+{-r + Xi)Pi{k-l) A; -2,3,4,... (23) 

In what follows, we summarize the findings about the auxiliary variable Pi{k) from eq.(15) 

and (23): 

A(l) = A, (24) 

A(2) = -rXi + {Xi-r)Xi (25) 

A(3) = r^\-r\(\-r) + {\-ryXi (26) 

; (27) 

A(^) = 5^ A,(-r)'=— (A,-r)— ^ (28) 

m=l 

= A,((-r)'=-i + (-r)'=-2(A,-r)^ 

+ (-r)'=-^(A,-r)^ + ...-r(A,-r)^-^ 

+ (A,-r)'=-^) (29) 



September 24, 2009 



DRAFT 



107 



Defining Q = Xi/r, we write the equation eq.(29) as follows 

m = r^a-if-^i - (G - 1) + (Ci - 1)' - ■ ■ ■ 

-(-1)^-1(0- 1)'^-^ 

+(-l)'=-^(C, - 1)^=-^) (30) 

Defining the sum S{k) 

S{k) = + 

(31) 

the eq.(30) is written as 

Pi{k) = r%S{k) (32) 
Summing S{k) with — l)S{k) yields S{k) as follows 

S(y = + (33) 

Si 

From eq.(32) and (33) 

P,{k) =r' + If) (34) 
From eq.(8), (22) and (34), the state transition matrix is 

= E^+i:i:ftWpv< (36) 

A:=0 k=Oi=l 

The first phrase of eq.(36) is equal to the exponential matrix series of —rtl, i.e., 



E = (37) 

fe=0 
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The second phrase in eq.(36), where the sums over i and k are interchangable, is obtained 
using eq.(34) as 



oo j.k N oo (_ri'\k 

EEAW^v. = (EE-4r^ + 

i=l k=0 i=l k=0 



^^(^i^_l}lLw, (38) 



i=lk=0 

N N 

= (E-e"''* + Ee''^'^"'^0^^ (39) 

i=l i=l 

So, let's put the eigenvalues into two sets: Let those eigenvalues which are smaller that r, 

belong to set T = {Aj^jj^Li where Nt is the length of the set; and let those eigenvalues which 
are larger than r belong to set S = {^js}fj=i where Ng is the length of the set. Using eqs. 
(37)-(39) and the definition Aj — Qr, we write the state transition matrix in eq.(35) as follows 

where 

Mtp{t) = e-^« - Yl e"'* V, + J2 e^^^' V,-* (41) 
j=i jteT 

and 

M.p(i) = E e^'^'^^'^^V,-^ (42) 

We call the matrices M.tp{t) and 'M.sp{t) in (41) and (42) as transitory phase part and steady 
phase part, respectively, of the state transition matrix. 

In eq.(41), because r > and A^^ — r < are finite numbers, the matrix Mtp(t) exponentially 
vanishes (approaches to zero matrix) and there exists a finite time constant tr^ for a given small 
positive number ei > such that 

\\Mtp{t>tT,)\\<e, (43) 
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where 1 1 ■ 1 1 shows a matrix norm. From eq. (40) and (43), the Mip(t) affects only the transitory 
phase, and what shapes the steady state behavior is merely M^p. So, let's examine the steady 
phase solution in the following: 

The steady phase solution, denoted as Xsp{t) is obtained from eq.(lO) and (40) as 

^sp{t) = M,p(t)x(0) = e("--'')*V,,x(0) (44) 

jses 

Let's define the interference vector 3sp{t) as 

3sp{t) = Wx,p(t) (45) 
Using eq.(ll) in (45) and the orthonormal features in (13) yields 



^sp{t) = E A,,e(^^-^-)*V,,x(0) (46) 
jses 

Defining Vj^x(O) = u^^, we rewrite the eq.(44) and (46), respectively, as 



x,p(t) = M,p(t)x(0) = J2 e(-'-+^-)*u,, (47) 

jses 

and 



Jsp{t) = E A,,e(-+^-)*u,, (48) 
jses 

In eq.(47) and (48), we assume that the Uj^ — Vj^x(O) corresponding to the eigenvector of the 
largest eigenvalue is different than zero vector. This means that we assume in the analysis here 
that x(0) is not completely perpendicular to the mentioned eigenvector. This is something easy 
to check in advance. If it is the case, then this can easily be overcome by introducing a small 
random number to x(0) so that it's not completely perpendicular to the mentioned eigenvector. 

Taking eq.(43) into account and dividing the vector 'x.sp{t) of eq.(47) to J(t) of eq.(48) 
elementwise and comparing the outcome with the "SIR" (9i{t)) definition in eq.(2) where bu — r 
results in 
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= -e,{tl t^oc, i^l,...,N (49) 
- -^^2^ ""^i^ (50) 



where Uj^(i) is the i'th element of the vector Wj^. From the analysis above, we observe that 

1) If there is only one positive eigenvalue and it's a multiple one, denoted as Ac (i.e. — 
Ac, js e -S"), then it's seen from (50) that 

Oi{t)^^, i^l,...,N, t>tT, (51) 

2) Similarly, if there is only one positive eigenvalue and it's a single one, shown as Ac, (i.e., 
5" = {Ac}), then eq.(51) holds. 

3) If there are more than two different (positive) eigenvalues and the largest positive eigen- 
value is single (not multiple), then we observe from (50) that the largest (positive) eigen- 
value dominates the dynamics of eq.(50) as time evolves because of the fact that a relatively 
small increase in the power of the exponential causes exponential increase as time evolves. 
This can be seen as follows: 

Let's show the two largest (positive) eigenvalues as X-max and Xj respectively and the 
difference between them as AA. So, Xmax — Xj + AA. We define the following ratio 



^ AA > (53) 



g(AA)t ' 

In eq.(53), because AA > is a finite number, the K{t) exponentially vanishes (approaches 
to zero), as will be depicted in Fig. 1, and there exists a finite time constant for a given 
small positive number 62 > such that 

\K{t>tT,)\<e2 (54) 
Similarly, for the denominator terms, we define the following ratio 
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\ {Xj-r)t A\ 

' = K{t){l - ^) < K{t) (55) 



AA, \j > 



The ratio between the exponentials of the highest two postitive eigenvalues with time 




time [seconds] 

Fig. 1. The figure shows the ratio K in eq.(52) for some different AA values. 



So, the K{t) in eq.(52) is obtained as a multiplier in the ratio of the two greatest terms 
of the sum in both nominator and denominator. We plot the ratio K{t) in Fig. 1 for some 
different AA values. The Figure 1 shows that the term related to the Xmax dominates the 
sum of the nominator and that of the denominator respectively. This observation implies 
from eq.(50) that 

xm_ _ E,..5e(-+^^-)^u,.(z) ^^^^ 



(57) 



Amaxe(-'-+^'--)* A..„„„ 
4) If the largest positive eigenvalue in case 3 above is a multiple eigenvale, then, similarly, 
the corresponding terms in the sum of the nominator and that of the demoninator become 

' // exponentially converges to -r 



dominant, which implies from eq.(50) that ' , / exponentially converges to t-^ time 



evolves. 
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Using the observations 1 to 4, and the "SIR" definition in eq.(2), we conclude from eq.(49), 
(56) and (57) that for any initial vector x(0) which is not completely perpendicular to the 
eigenvector corresponding to the largest eigenvalue of W, 

e,{t^^) = -^^-—, i = l,...,Ar, (58) 

which completes the first part of the proof. Furthermore, from eq. (43), (54) and (58) we 
conclude that there exists a finite time constant tr for a given small positive number €c> such 
that 



1^"" (t > tr) - < ec (59) 

where O^^^it) = [ei{t) . . . 0N{t)f and e""^^'"^" = . . . 1]^, which completes the proof. 

■ 

Definition: System-Specific Ultimate SIR value: In proposition 1, we showed that the SIR in 
(2) for every state in the autonomous linear dynamic networks in eq.(4) converges to a constant 
value as time goes to infinity. We call this converged constant value as "system specific ultimate 
SIR" and denote as 

0Ult ^ _J_ (gQ) 

■^max 

where r > is the design parameter and Xmax is the maximum (positive) eigenvalue of matrix 
W. 



B. Discrete-time Analysis 

In this subsection, we analyse the defined "SIR" for the the underlying linear part of the 
proposed discrete-time autonomous network which is obtained by discretizing the continuous- 
time system of eq.(4) by using well-known Euler method: 

x(A; + 1) = (I + a(-rI + W))x(A;) (61) 

where I is the identity matrix, r is a positive real number, — rl and W is as eq.(5), x(A;) 
shows the state vector at step k, and a > is the step size. 



September 24, 2009 



DRAFT 



113 



Proposition 2: 

In the autonomous discrete-time Unear network of eq.(61), let's assume that the spectral radius 
of the system matrix (I + a{—rl + W)) is larger than 1. (This assumption is equal to the 
assumption that W has positive eigenvalue(s) and r > is chosen such that Xmax > where 
Xmax is the maximum (positive) eigenvalue of W). If a is chosen such that < ar < 1, then 

1) the defined "SIR" 9i{k) in eq.(2) for any state i asymptotically converges to the "ultimate 
SIR" constant in (6) as time step evolves for any initial vector x(0) which is not completely 
perpendicular to the eigenvector corresponding to the largest eigenvalue of W. 

2) there exists a finite step number kr for a given small positive number > such that 

||r""(A; > A;t) -^''"'^'11 < ed (62) 
where O'^^'^ik) = [^i(A;) . . . eN{k)f and r'*'"^^ = . . . 1]^. 
Proof: 

From eq. (61), it's obtained 

x{k) = (l + a{-rl + W))^x(0) (63) 

where x(0) shows the initial state vector at step zero. Let us first examine the powers of the 
matrix (l + a{—rl + W)^ in (63) in terms of matrix rl and the eigenvectors of matrix W: It's 
well known that any symmetric real square matrix can be decomposed into 

N N 

W = ^A,v,vf = ^A,V, (64) 

i=l i=l 

where {Xij^Li {^i}iLi show the (real) eigenvalues and the corresponding eigenvectors 
and the eigenvectors {vi}fLi are orthonormal (see e.g. [16]), i.e.. 



v/ V, = <; (65) 



1, if i = j, where i,j = l,2,...,N 
0, if i^j, 

Let's define the outer-product matrices of the eigenvectors {Xij^i as = Vjvf , i 
1,2, ... ,N; and, furthermore, the matrix M as 
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M = I + a(-rI + W) 
which is obtained using eq.(64) as 



N 



M = (l-ar)I + ^A(l)Vi 

i=l 

where r > 0, a > 0, and where Pi{l) is equal to 

A(l) = aXi. 

The matrix can be written as 

N 

M2 = (l-ar)2l + ^A(2)V, 
1=1 

where Pi{2) is equal to 

l3i{2) = a{l - ar)Xi + (1 - ar + aXi)(3i{l) 
Similarly, the matrix can be written as 

N 



M3 = (l-ar)='l + ^A(3)V, 



1=1 



where /3i{3) is equal to 



A(3) = a(l - arfXi + (1 - ar + aAi)A(2) 
So, can be written as 



N 



M^ = (l-ar)^I + $:A(4)V, 



1=1 



where /3i(4) is equal to 



A(4) = a(l - ar)=^Ai + {1 - ar + q;\)A(3) 
So, at step k, the matrix (M)'^ is obtained as 
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i=l 



where Pi{k) is equal to 



Pi{k) = a{l - ar)''-'Xi + (1 + a{Xi - r))pi{k - 1) 
Using eq.(68) and (76), the Pi{k) is obtained as 

A(2) = aA,((l-ar) + (l + a(\-r))) 
A(3) = Q;Ai((l-o;r)' + (l-Q;r)(l + Q;(Ai-r)) 
+ (l + «(A,-r))') 



= aA, ^(l-ar)'^"'"(l + a(A,-r)r-' 
Defining Aj = ~ Q;^)> we obtain 



(1 - ar)'=-™(l + a(Ai - r))'"-' = (1 - ar)''-\l + aQf 
Writing eq.(81) in eq.(80) gives 

m^aCi{l-ar)''S{k) 

where S{k) is 

k 

sik)^j:ii+aCir'' 

m=l 

Summing -S{k) with {l + aCi)S{k) yields 

ay 

From eq.(82), (83) and (84), we obtain 
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Pi{k) = (1 - ar)^! + aQ' - (1 - ar f (85) 
Using the definition Q = A^/ (1 — ar) in eq.(85) gives 

(3i{k) = (1 + a{Xi - r))" - (1 - ar)' (86) 

From eq.(75) and eq.(86), 

TV TV 

M'^ = (1 - ar)''I + 13 (1 + a{Xi - r)) V, - ^(1 - ar)'=V, (87) 

i=l 1=1 

Let's put the N eigenvalues of matrix W into two groups as follows: Let those eigenvalues 
which are smaller that r, belong to set T = {Ajjj^Li where Nt is the length of the set; and let 
those eigenvalues which are larger than r belong to set S = {Xj^}ff^i where Ng is the length 
of the set. We write the matrix M*^ in eq.(87) using this eigenvalue grouping 

M*^ = Mtp{k) + M,p{k) (88) 

where 

TV 

Mtp{k) = (l-ar)'=I-^(l-ar)'=V, 

i=l 

+ E (1 + «(A,. - r))'Vj, (89) 

jteT 

and 

M,,(^)= 5:(l + a(A,-,-r))'=V,-, (90) 

jses 

We call the matrices 'Mtp{k) and 'M.sp{k) in (89) and (90) as transitory phase part and steady 
phase part, respectively, of the matrix M'^. 

It's observed from eq.(89) that the 'Mtp{k) converges to zero matrix as time step number 
evolves because relatively small step number a > is chosen such that (1 — ar) < 1 and 
1 + a{Xj^ — r) < 1. Therefore, there exists a finite time step number kx^ for a given small 
positive number €3 > such that 
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\\Mtp{k>kT,)\\<^3 (91) 

Thus, (from eq.(66) and (88)), what shapes the steady state behavior of the system in eq.(63) 
is merely the Msp(fc) in eq.(90 ). So, the steady phase solution is obtained from eqs.(63), (66) 
and (88)-(90) using the above observations as follows 

^sp{k) = M,p(A;)x(0) (92) 
= E(l + "(^..-O)'V,.x(0) (93) 

Let's define the interference vector, Jsp{k) as 

J,p{k) = W^,p{k) (94) 
Using eq.(64) in (94) and the orthonormal features in (65) yields 

Jsp{k) = E ^..(1 + '^(K - O)'V,.x(0) (95) 
jses 

First defining Vjx(O) = Uj, and ^ = T^-> ^^^^ dividing vector Xgpik) of eq.(93) to 3sp{k) of 
eq.(95) elementwise and comparing the outcome with the "SIR" definition in eq.(2) results in 



"^^^'^^^^ - -e,ik), i^l,...,N (96) 



Jsp,i{k^ ^ 

In eq.(97), we assume that the Wj = Vjx(O) which corresponds to the eigenvector of the largest 
positive eigenvalue is different than zero vector. This means that we assume in the analysis here 
that x(0) is not completely perpendicular to the mentioned eigenvector. 

From the analysis above, we observe that 

1) If there is only one positive eigenvalue which is greater than r and it's a multiple one, 
denoted as A^, then it's seen from (97) that 



e.(A;) = -^, A;^oo, i = l,...,A^ (98) 
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2) Similarly, if there is only one positive eigenvalue which is larger than r and it's a single 
one, shown as A^, then eq.(98) holds. 

3) If there are more than two different (positive) eigenvalues and the largest positive eigen- 
value is single (not multiple), then we see from (96) that the term related to the largest 
(positive) eigenvalue dominates the sum of the nominator. Same observation is valid for the 
sum of the denominator. This is because a relatively small increase in Xj causes exponential 
increase as time step evolves, which is shown in the following: Let's show the two largest 
(positive) eigenvalues as Xmax and Xj respectively and the difference between them as AA. 
So, Xmax — Aj + AA. Let's define the following ratio between the terms related to the two 
highest eigenvalues in the nominator 

where 

e = (100) 

I — ar 

In eq.(99), because A A > 0, there exists a finite time step number for a given small 
positive number 63 > such that 

\K^{k)\<es (101) 

Similarly, let's define the ratio between the terms related to the two highest eigenvalues 
in the denominator as 

= , ..... (102) 



From eq.(99) and (102), because AA > and < 1, 



(A, + AA)(l + e(A, + AA))^ 

A,+AA 



Ka{k) < K^{k). (103) 

and there exists a finite time step number for a given small positive number 64 > 
such that 
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\Kn{k > /EtJI < 64- 



(104) 



The ratio between the exponentials of the highest two postitive eigenvalues with time 
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step number k 



— deltaL=0.5 
— deltaL=1 

deltaL=2 
-< — deltaL=3 

deltaL=5 



140 160 



180 



Fig. 2. The figure shows the ratio Kn in eq.(99) for some different AA values (A = 5,^ = 0.11). 

We plot the ratio Kn{k) in Fig. 2 for some different AA values and for a typical ^ value. 
The Figure 2 and eq.(103) implies that the terms related to the Xmax dominate the sum of 
the nominator and that of the denominator respectively. So, from eq.(97) and (100), 



X 



sp,i 



{k) 



u 



J sp,i{k^ 



max } 



1 



(105) 



4) If the largest positive eigenvalue in case 3 above is a multiple eigenvale, then, similarly, 
the corresponding terms in the sum of the nominator and that of the demoninator become 
dominant, which implies from eq.(97), (99) and (102) that converges to as 

step number increases. 



Using the observations 1 to 4, eq.(91), the "SIR" definition in eq.(2), eq.(96) and (97), we 
conclude that 



e^ik 



oo 



(106) 
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where i = 1, . . . ,iV, and Xmax is the largest (positive) eigenvalue of the matrix W, which 
completes the first part of the proof. Furthermore, from eq. (91), (104) and (106), we conclude 
that there exists a finite time constant kr for a given small positive number > such that 

I |r"^(A; > kr) - r'*'"^^! I < ea (107) 
where O^'^^ik) = [9i{k) . . . 9N{k)f and = . . . 1]^, which completes the proof. 

■ 

III. Stabilized SIR system 

Do the results of the ultimate SIR analysis in Section II above have any practical meanings? 
Our answer is yes. In this section, we propose two Hopfield-like networks in continuous and 
discrete-time domain respectively where the "system-specific ultimate SIR" is used to stabilize 
the system. 

A. Continuous -time SALU-"SIR" 
Defining the following g{a) function. 



9(a) = < 

we propose the following dynamic network 



1 if \a\ > e, 

' ' ~ (108) 
otherwise 



X = (-rl + W)x^(||r'=^(t) -^"'^'='=11) (109) 
y = sign(-x.) (110) 

where O^^^t) = [^i(t) . . . ^jv(t)]^ and r"'''^'^ = . . . 1]^, and y is the output of the 
network. 

We call the network in eqs.(108)-(l 10) as Stabilized Autonomous Linear Networks by Ultimate 
"SIR" (SAL-U"SIR"). 

Proposition 3: 
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Checking 



Fig. 3. The figure depicts a sketch of the proposed network for numerical implementation purposes, omitting the considerations 
on the circuit, called Stabilized Autonomous Linear Networks with Ultimate "SIR" (SAL-U"SIR"). 

The proposed dynamic network of eqs.(109)-(l 10) with the weight matrix (— rl + W) as 
defined in eq. (5) is stable for any initial condition x(0). 



If all the eigenvalues of the symmetrix matrix (— rl + W) as defined in eq. (5) are negative, 
then it's well known from linear systems theory that the states go to zero exponentially for any 
initial vector x(0) (see e.g. [17]). 

Otherwise, if there exists positive eigenvalue(s) of (— rl + W), which is the case in our design, 
then the proposition 1 above proves for the underlying linear system of (109) that i) the defined 
SIR in eq.(2) for any state i (Xj(t), 1, 2, ... , A^,) converges, as time evolves, to the constant 
system-specific ultimate SIR value in eq.(6) for any initial vector x(0), and ii) there exists a finite 
time constant tr for a given small positive number e > such that | \9^'^'^(t > tx) — 0'^'-^''"^^\ \ < e. 
So, the function g{-) stabilizes the system within the seconds, i.e., once | \0^^^(t)~6^^*''"'^'^\ | < e 
is met. So, the system is stable. 

■ 

From the analysis above for symmetric W and positive rl, we observe that 

1) The SAL-U"SIR" does not show oscilatory behaviour because all eigenvalues are real (i.e., 
no imaginary part), and at least one eigenvalue is positive, which is assured by choosing 
the matrix rl accordingly. 

2) The transition phase of the "unstable" linear network is shaped by the initial state vector 



Proof: 
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and the phase space characteristics formed by the eigenvectors. The network is stabilized 
by the function g{-) once the network has passed the transition phase. The output of the 
network then is formed taking the sign of the converged states. Whether the state converges 
to a plus or minus value is dictated by the phase space shaped by the eigenvectors of W 
and the initial vector x(0). 
From observation 1 and 2 above, we see that choosing the positive matrix rl such that the 
matrix (— rl + W) has positive eigenvalues makes the proposed SAL-U"SIR" exhibit similar 
features as the Hopfield Network does. The computer simulations in section IV shows the 
performance of the proposed network as compared to the Hopfield Network in some simple 
associative memory systems examples. 

As far as the possible circuital implementations of the proposed network is concerned, a further 
research is needed on especially how the nonlinear function g(-) could be implemented on the 
circuit. In this paper, we mainly focus on the analysis part, and the circuital implementation 
part is left as a future research topic. However, in order to give an insight, in what follows, we 
propose the following simplified network for numerical implementation purposes: 

It's well known that a linear dynamic network like (4) can be implemented by RC (Resistance- 
Capacitance) circuits. Corresponding RC dynamic network is given as follows: 



CVc, 




Rii 


R12 ■ 


■ RlN 






CVc, 




R21 


. 


■ R2N 






CVc^ _ 




Rni 


Rn2 ■ 


. 







(Ill) 



where C represents the capacitance, shows the voltage of the capacitance Q, which is 
the state i of the network and Rij is the resistance. From eq.(l), (5) and (111), 



Rij = -^Wij, i,j = 1,2,...,N and i j, (112) 

Rii = r (113) 

So, we sketch a simplified numerical implementation of the proposed SAL-U"SIR" in Fig. 
3, omitting the considerations on circuits, where the function g{-) is represented by a switch 
("ultimate SIR checking"). 
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B. Discrete-time SALU-"SIR" 

The proposed autonomous network in discrete-time, called DSAL-USIR (Discrete Stabilized 
Autonomous Linear networks by Ultimate "SIR") is given as follows 

x(A; + l) = (l + o;(-rI + W))x(A;)y(||r""(i) (114) 
y{k) = sign{^{k)) (115) 

where W is defined by eq.(5), the function g(-) is defined by eq.(108), a > is step size, I 
is identity matrix and r > as in eq.(61), 9""%t) = [^i(t) . . . eN{t)f and ^""'''^'^ = ^"'*[1 . . . 1]^, 
and y{k) is the output of the network. 

Proposition 4: 

The proposed discrete-time networks, DSAL-U"S1R", in eq.(114) and (115) is stable for any 
initial vector x(0). 

Proof: 

If the spectral radius of the system matrix (I -|- a{—rl + W)) is smaller than 1, then it's well 
known fi-om the discrete-time linear systems theory that the states go to zero exponentially for 
any initial vector x(0) (see e.g. [17]). 

If, on the other hand, the spectral radius is larger than 1, which is the case in our design, 
then the proposition 2 above proves for the underlying linear system (61) that i) the defined 
"SIR" 9i{k) in eq.(2) for any state i asymptotically converges to the "system-specific ultimate 
SIR" constant in (6) as time step evolves for any initial vector x(0) ii) there exists a finite step 
number kr for a given small positive number > such that \\9'"^'^{k > kx) — < e^. 

So, the function g(-) stabilizes the system within the /ct steps, i.e., once ||6'^^'^(t) _5)"^*'^ec|| ^ 
is met. So, the system is stable. 

■ 

As far as the design of weight matrix W and r is concerned, we propose to use the following 
method which is based on the well known Hebb-leaming rule [15]. 
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C. Outer products based network design 

Let's assume that L desired prototype vectors, {ds}fLi, are given from (—1,+!)^. The 
proposed method is based on well-known Hebb-leaming [15] as follows: 

Step 1: Calculate the sum of outer products of the prototype vectors (Hebb Rule, [15]) 

Q = X^d,df (116) 

s=l 

Step 2: Determine the diagonal matrix rl and W as follows: 

r^qa + p (117) 

where p is a real number and 

if i = j, 

i,j^l,...,N (118) 

Qij if 



Wij 



where shows the entries of matrix Q, N is the dimension of the vector x{t) and L is the 
number of the prototype vectors (N > L > 0). From (116), qa — L in eq.(117) since {d^} is 
from (-1,+!)^. 

If the desired prototype vectors are orthogonal, then it can be shown, using the steps of the 
proofs of prepositions 1 and 3 for continuous and discrete-time respectively, that the "system 
specific ultimate SIR" be = 

IV. Simulation Results 

We take the same examples as in [14] for comparison reasons and for the sake of brevity. 

In this section, we apply the the proposed networks SAL-U"SIR" and DSAL-U"SIR", in 
continuous and discrete-time respectively, to associate memory systems design, and present their 
simulation results as compared to those of corresponding Hopfield Networks. The weight matrices 
of the proposed networks and the Hopfield Networks are designed by the outer-products (Hebb 
learning [15]) learning rule described in Section III-C. 
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A. Continuous-time examples 

In this section, we present two examples, one with 8 neurons and one with 16 neurons, in 
Example 1 and 2 respectively. 

The proposed DSALU-SIR network is given by eqs. (109) and (110). The Hopfield Network 
[1], used as the reference network, is given by 



X = -rx + Wf(x(i)) + b (119) 
y{t) = f(x(t)) (120) 

where W is the weight matrix and x(i) is the state at time b = 0, f (x) = [f{xi)f{x2) ■ ■ ■ f{xM)Y, 
the /(•) is a sigmoid function, i.e., f\xi) = 1 - iip^^^p^, where tr > 0. 

Example 1: 

This example is taken from example 1 in [14]. In the design, u = 2 and p is chosen as -2, 
and r = 1. The desired prototype vectors are given in the raws of matrix D as follows, 



D 



1 1 1 

1 1 -1 
1-11 



1 -1 -1 -1 -1 

-11 1-1 -1 
-11-11 -1 



(121) 



The weight matrix, using the design rule in Section ni-C, is obtained as 



W 







rl 


= 1, 




and 
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(122) 



(123) 
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The Figure 4 shows the percentages of correctly recovered desired patterns for all possible 
initial conditions x(t = 0) G (—1, +1)^, in the proposed SALU-"SIR" as compared to traditional 
Hopfield network. 



Percentage of correctly recovered prototype vectors, N=8 




Hopfield Network SAL-U"SIR" 



Fig. 4. The figure shows percentage of correctly recovered desired patterns for all possible initial conditions in example 1 for 
the proposed SALU-"SIR" as compared to traditional Hopfield network with 8 neurons. 

Let TUd show the number of prototype vectors and C{N, K), (such that N > K > Qi), represent 
the combination N,K, which is equal to C{N,K) = jji^zj^jfj^, vvhere ! shows factorial. In our 
simulation, the prototype vectors are from (—1,1)^ as seen above. For initial conditions, we 
alter the sign of K states where K=0, 1, 2, 3 and 4, which means the initial condition is within 
i^-Hamming distance from the corresponding prototype vector. So, the total number of different 
possible combinations for the initial conditions for this example is 24, 84 and 168 for 1, 2 and 
3-Hamming distance cases respectively, which could be calculated by rud x C{8,K), where 
md = 3 and = 1, 2 and 3. 

As seen from Figure 4, the performance of the proposed network SALU"SIR" is the same as 
that of the continuous Hopfield Network for 1 -Hamming distance case (%100 for both networks) 
and is slightly higher than that of the Hopfield Network for 2 distance case. 

Example 2: 

This example is taken from example 2 in [14]. The desired prototype vectors as well as the 
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obtained weight matrix W are shown in Appendix I. The other network paramaters are chosen 
as in example 1: a = 2 and p = —2 . 

The Figure 5 shows the percentages of correctly recovered desired patterns for all possible 
initial conditions x(t = 0) G (—1,+!)^^, in the proposed SALU"SIR" as compared to the 
traditional Hopfield network. 



Percentage of correctly recovered prototype vectors, N=16 




Hopfield Network SAL-U"SIR" 



Fig. 5. The figure shows percentage of correctly recovered desired patterns for all possible initial conditions in example 2 for 
the proposed SALU-"SIR" as compared to traditional Hopfield network with 16 neurons. 

The total number of different possible combinations for the initial conditions for this example 
is 64, 480 and 2240 and 7280 for 1, 2, 3 and 4-Hamming distance cases respectively, which 
could be calculated by m^^ x C(16, K), where = 4 and K = 1, 2, 3 and 4. 

As seen from Figure 5 the performance of the proposed network SALU"SIR" is the same as 
that of Hopfield Network for 1, 2 and 3-Hamming distance cases (%100 for both networks), and 
gives comparable performance with the Hopfield Network for 4-Hamming distance case. 

B. Discrete-time examples 

In this section, we present two examples, one with 8 neurons (Example 3) and one with 16 
neurons (Example 4). The traditional discrete Hopfield network [1], shown in the following, is 
used as a reference network: 
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x'=+^ = si^n(Wx^) (124) 

where W is the weight matrix and x'^ is the state at time k, and at most one state is updated 

at a step. 

In the simulations in this subsection, we also examine the following version of the DSAL- 
U"SIR" for comparison reasons: 

x(A; + l) = {-pI + W)x{k)g{\e'"''{t) -e^^^'^^^l) (125) 
y(A;) = signi^ik)) (126) 

where I is the identity matrix, 1 > p > and W is defined in eq.(5), and y(A;) is the output of 
the network. It can be shown that the above network is stable using the steps in DSAL-U"SIR" 
in previous section. Here, we omit the proof for the sake of brevity and present only the results 
for comparison reasons. 

Let's denote the original network in eqs.(114) - (115) as DSAL-U"SIR"1, and let's call the 
network in eq.(125)-(126) as DSAL-U"SIR"2. 

Example 3: 

The desired prototype vectors are given in the raws of the following matrix 



D 



(127) 



111 1-1-1-1-1 
11-1-11 1-1-1 
The weight matrices rl and W, and the threshold vector b are obtained as follows by using the 
outer-products-based design mentioned above and p is chosen as -1 and for the DSALU-U"SIR"2 
network, p = 0.5. 
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rl = 21, (128) 
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d = (130) 

The Figure 6 shows the percentages of correctly recovered desired patterns for all possible 
initial conditions x(A; = 0) G (-1, +1)^, in the proposed DSALU-"SIR"1 and DSALU-"SIR"2 
as compared to the traditional discrete Hop field network in (124). 



Percentage of correctly recovered prototype vectors, N=8 




Discrete Hopfield Network DSALU-"SIR"1 DSAHJ-"SIR"2 



Fig. 6. The figure shows percentage of correctly recovered desired patterns for all possible initial conditions in example 3 for 
the proposed DSALU-"SIR"1 and DSALU-"SIR"2 as compared to traditional discrete Hopfield network with 8 neurons. 



As seen from Figure 6, the performances of the DSALU-"SIR"1 and 2 are the same as that 
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of the discrete-time Hopfield Network for 1 -Hamming distance case (%100 for both networks) 
and are comparable for 2 and 3-Hamming distance cases respectively. 

Example 4: 

The desired prototype vectors as well as the obtained weight matrices are given in in Appendix 
II (eq.(134)). 

For matrix rl, p is chosen as -2. The other network paramaters are chosen as in example 3. 

The Figure 7 shows the percentages of correctly recovered desired patterns for all possible 
initial conditions x(A; = 0) G (—1, +1)^^, in the proposed DSALU"SIR"1 and 2 as compared to 
the traditional Hopfield network. 



Percentage of correctly recovered prototype vectors, N=16 




Discrete Hopfield Network DSALU-"SIR"1 DSALU-"SIR"2 



Fig. 7. The figure shows percentage of correctly recovered desired patterns for all possible initial conditions in example 4 for 
the proposed DSALU-"SIR"1 and DSALU-"SIR"2 as compared to the traditional discrete Hopfield network with 16 neurons. 

The total number of different possible combinations for the initial conditions for this example 
is 64, 480 and 2240 and 7280 for 1, 2, 3 and 4-Hamming distance cases respectively. 

As seen from Figure 7 the performance of the proposed networks DSALU-"SIR"1 and 2 are 
the same as that of Hopfield Network for 1 and 2-Hamming distance cases (%100 for both 
networks), and are comparable for 3,4 and 5-Hamming distance cases respectively. 
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V. Conclusions 

In this paper, we present and analyse two Hopfield-like nonlinear networks, in continuous-time 
and discrete-time respectively. The proposed network is based on an autonomous linear system 
with a symmetric weight matrix, which is designed to be unstable, and a nonlinear function 
stabilizing the whole network thanks to a manipulated state variable. This variable is observed 
to be equal to the traditional Signal-to-Interference Ratio (SIR) definition in telecommunications 
engineering. 

The underlying linear system of the proposed continuous-time network is x = Bx where B is 
a real symmetric matrix whose diagonal elements are fixed to a constant. The nonlinear function, 
on the other hand, is based on the defined system variables called "SIR"s. We also show that 
the "SIR"s of all the states converge to a constant value, called "system-specific Ultimate SIR"; 
which is equal to where r is the diagonal element of matrix B and X-max is the maximum 
(positive) eigenvalue of diagonally-zero matrix (B — rl), where I denotes the identity matrix. 
The same result is obtained in its discrete-time version as well. 

Computer simulations for binary associative memory design problem show the effectiveness 
of the proposed network as compared to the traditional Hopfield Networks. 
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Appendix I 

In Example 2, the matrix which has the desired prototype vectors as its raws is 
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(131) 

In Example 2, the weight matrices rl and W, which are obtained by the outer products based 
design as explained in Section III-C, are as follows: 
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rl = 21 (132) 
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Appendix II 

In Example 4, the matrix which has the desired prototype vectors as its raws is 



D = 



11111111 
111 1-1-1-1-1 
11-1-11 1-1-1 



-1 -1 -1 -1 -1 -1 -1 -1 
1 1 1 1-1-1-1-1 
1 1-1-11 1-1-1 



(134) 



In Example 4, the weight matrices rl and W obtained are as follows: 
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